In a world where AI delivers instant answers and infinite information, the challenge isn’t scarcity, it’s overload.
How do we navigate a landscape where shortcuts abound, attention is fragmented and the very meaning of “knowing” is being redefined?
In this Davos town hall, leaders from Cambridge, Udemy and Cohere argued that AI has shifted the dilemma from access to knowledge to the capabilities needed to use it well. The audience ranked “independent judgment and critical thinking” and “sustained human attention” as the scarcest resources, echoing Hugo Sarrazin’s reminder that “when you have a wealth of information, you have a poverty of attention.” Aidan Gomez warned that chatbots can create “a false sense of mastery,” making rigorous assessment essential: “You need to take away the tool and see what the human alone understands.” Both executives emphasized AI’s upside as scalable personalization, revisiting Bloom’s “two sigma” effect by approximating one-on-one tutoring through adaptive feedback loops, simulation and role play. The panel also surfaced a looming organizational risk: seniors may be able to judge AI output, but if juniors are displaced, firms may undermine the next generation’s ability to QA AI-driven work. On authority and trust, Sarrazin predicted a swing from “every Reddit quote” training to “specialized trusted” models, while Gomez pointed to reasoning models and retrieval-augmented generation that can “cite directly back” to authoritative sources. The session closed on whether the university degree remains a valuable “bundle” of learning, accreditation and rite of passage—or is ripe for unbundling as AI changes delivery economics.
Good afternoon, everyone, and thank you for joining this town hall discussion, where we will be talking about a topic that university and education leaders are all buzzing about, which is, namely dilemmas around knowledge. This has been a topic for us since schools were first invented. Libraries were first invented. And it's still with us today. It's extremely relevant today in an age in which AI is changing, making knowledge available broadly to everybody all the time. But it doesn't mean that there aren't still dilemmas around knowledge. And we're going to probe these today. I'm Professor Debbie Prentice, and I'm the vice chancellor of the University of Cambridge. I'm very pleased to introduce you to our panelists for this session. So we have Aidan Gomez, who is the co-founder and chief executive officer of cohere, an enterprise AI company developing advanced language models for use by business. And we also welcome Hugo Sarrazin, who is president and chief executive officer of Udemy, which provides a wide range of business and leadership development courses, including AI courses to businesses and organisations around the world in fields such as financial services, higher education, government, manufacturing and technology. We have some fascinating questions to discuss this afternoon around knowledge, misinformation, AI, attention spans and even the nature of expertise. And we're going to bring the audience in, early and often. So I hope that you'll all participate with us. We as panellists, come from very different perspectives. Aidan and Hugo run a very successful businesses selling a product. They are from the for profit educational technology sector. And I'm from the not for profit sector. So there are different pressures, different opportunities, different challenges that we face in this space. Before we get started with our panel discussion, I'd like to remind the online audience, that, if you are sharing with us through your social channels, they should use the hashtag. You should use the hashtag, hashtag F26. And whether you are joining online today or here in person. And it's great to see so many of you here. Thank you so much for coming. Please feel free to get involved in the session by reacting to the questions we discussed in our conversation. And also by submitting questions to panelists via the Slido app. Okay. Okay. So our first question is, in a world of instant answers and AI assistants, what is becoming the Scarcest resource? Okay. The answers are from a list of options. Is it sustained human attention, independent judgment and critical thinking, deep understanding and mastery. Motivation to learn in the first place, or trust in what we know and who to believe? And actually I said, or that could be and you can choose as many of these as you as you want. Okay. So you can see on the screen, actually, as people are responding via the Slido, app, but I want to ask our panelists, what would you say? So you can see the answers on the screen. What would you say, Hugo?
Well, I think it's a complicated question, and I think there's a lot of all of the above. If you take a historical perspective. Knowledge was scarce. That was a source of power. You know, countries fought for that. And we also had experts that built knowledge over time. But very few polymaths, very few those ones that were were very, very, very important. Now today you have llms that can learn everything and they can learn across different domains and they can become the polymaths. So every data center, every time we say there's a new infrastructure that's being added, we're we're adding millions and millions of polymaths, and that becomes a democratization of that knowledge. The problem is, and, you know, there's there's an amazing quote from Herbert Simon, when you have a wealth of information, you have a poverty of attention. And I think that's what's happening for a lot of learners. And that's why traditional methods need to change. And we're going to come up and talk, I'm sure, about how learning needs to evolve, what the process, what's the role of traditional institution and changing, what's the role corporations need to and what individual needs to do. So I think attention is one big component. The second is a lot of when you go to LLM and AI and you ask for a question, it will give you an answer. It will feel very comfortable with that answer. It doesn't explain explainability in AI is a whole field, a whole domain. And most of these llms don't give you that. So if you have a society that begins to rely on products that give you an answer, but don't tell you where that answer came from, how do you learn and what do you have in terms of trust? So I think the trust piece is also equally important. So I'll stop at that. We can go further.
But yeah, I was looking at the the poll up there and for whatever reason, the first one that came to me was Deep Mastery, which seems to be the most unpopular choice for whatever reason. So I think, you know, when you exist in a world where it's so fast and easy to get answers to whatever question you might have, or to get a very surface level answer to even a complex question like, whatever, how does quantum mechanics work? It'll give you a four paragraph response. But that's not deep understanding of the subject matter. And so I think Llms can't chatbots, they can fool you into thinking that you understand something when you don't. And I view that as a core risk as we integrate these llms into an education environment. Is this false sense of mastery or understanding? You know, we can discuss the different solutions to that. I think that testing is essential to it. The idea that you need to take away the tool and see what the human alone understands and has retained, the ability for you to assess depth has to take away those tools. And I think that is, from my perspective, what's most at risk.
Yeah. It's interesting. You know, my, my, my answer is a variant on yours. I wanted to I, of course, wanted to reject all five. But I think it's because I think it's because of where I come from. Coming from the university sector. I mean, I wanted to say self-knowledge of the for the learner. Right. And it's it's part of what you're saying. I mean, you don't know if you've mastered it and you don't know if you're interested in it, and you don't know if you get it right. It comes to you so much of what you learn, so much of what you learn comes from what is difficult and, and what is compelling and, and what. So for that, for those cues to no longer be actually, you know, useful cues for self-understanding means how will you even know? But that's my answer anyway. So we can see what the oops, I it went away. I think critical thinking was the one that won out at the end. It looked like critical thinking was was actually the the audience preferred. We can we can keep coming back to this. But I want to I want to use this as a jumping off. Oh there we go. Okay. Yeah. Critical thinking and then sustained attention. They were they were neck and neck for the most of the time. Yeah. And then trust and then deep deep mastery. Right. It's interesting. So I want to talk a little bit about, about each of, what you do. So we could start with you, Hugo. You know, tell us about Udemy.
So Udemy is a 15 year old company that at the time did a pretty cool thing around introducing online learning. It was a great innovation. It changed accessibility and the cost of reaching out to millions and millions of people and created a creator economy around that. So we now have 250,000 courses, 80 million learners on a regular basis. We serve 17,000 large enterprise. We have 85,000 instructors that kind of come to this marketplace to offer their where, they're very deeply committed. They know stuff and they want to share it to the world. And we do it in, you know, about 40% of our, revenues are in the US. The rest is around the world. So we're in tons of languages, 46 plus. And, the funny story, I've only been in the role for less than a year. When I came in my first town hall. And the people who may be listening online who were on that town hall, I came in and I said, we're going to exit online learning. That is a wonderful innovation. It did a bunch of great things, but it doesn't solve the problem of today. And with AI we can do so many different things. So I want to make a hard pivot of the business toward becoming an AI platform to reskill the workforce of the future. And we can talk about that. And I don't want to take too much time, but there's a lot of, you know, ways you can use AI to do some of the things you're suggesting to kind of help build the mastery. How to do assessment using AI, how to use AI, role play to immerse people. And it also does a thing that I think is so, so important, traditional online learning and actually traditional learning. You you're an instructor and you teach to the average, right? You create your curriculum and you think you're going to hit the most of the people you can't get for the super fast, you can't get for the super slow, and the same on online learning. And then different people have different starting points, and we don't have an easy way to accommodate that. Now with AI, you can do a quick assessment. You can break apart the class. You can have feedback loop and reinforce that in a very, very powerful way. And I think that's one of the things that's going to emerge of using AI to kind of reskill. The workforce is going to build on that previous generation of online learning to do something pretty remarkable and quite different moving forward.
Thank you. Aidan.
Yes, sir. Cahir builds large language models. So we're one of the developers of this core piece of technology, that powers things like ChatGPT and all these different applications. We're focused purely on the enterprise side of house, and so we work with businesses to put those models to work inside the organization. We give them access to internal data and systems that the humans have access to. And then we teach or we work with our customer to teach the workforce, to shift their role from being the ones individually doing the work to managing a team of these models or agents to carry out that work. Our big differentiator is on the security side, so there's no data exiting our customers perimeter. Instead, we send all of our models and software to them, and they keep it self-contained.
Yeah. So so you must be you have certain customers who will only subscribe to you, right?
Yeah. It certainly critical industries, financial services, telco, healthcare, and then of course government applications as well. Anything that's a national security concern. And arguably education is within that remit. That's a place that we do extremely well.
That's interesting. So Hugo, what can we learn from the the arc of progress from MOOCs and online education to now AI driven?
I think a few things. The first one is, you know, if you look at the traditional learning processes and methods that we had, there was a void. And that's why online learning took off, and that's why there's a whole industry. And it, addressed a bunch of problems around, you know, getting to skills, specific skills and also getting to certification and then helping organizations reskill so that that that was a very, very, very powerful thing. What is now becoming a lot more, a priority. And in the last, six months, I spend an enormous amount of time, I spoke to 400 and head of learning and development in large enterprise. So the pattern that I saw is they had an enormous proliferation of tools and things that were bought during the Covid era. Very few could explain the ROI. What's the how do you measure the ROI of learning? It's a really good question. And everybody kind of defaulted to did they take the class? Did they complete the class hours of learning? And as a business leader, it's not particularly helpful. And it gets even worse when they get certification in Google Cloud or AWS or cyber or something to know that you've certified yourself two years ago. I'm a business leader. I want to know, are you current? Are you relevant today? So I think the arc now is moving and the enterprise to an ability to do in the flow of work learning. Do it at bite size, do it in an adaptive way, and we can come back to what adaptive means. And with an ROI, an ability to measure what, you know, skills people are deploying in real time. So you're now beginning to create a workforce management tool that is powered by an operating learning system.
So, Aidan, you said that you said that you were not as worried about sustained human attention as you were some of the others. How does cohere solve the attention problem?
Well, I mean, I don't know if cohere solves the attention problem. I think it it's definitely a concern. There's lots of pressures on our attention span. I think, social media, short form content is, driving a lot of that. I'm certainly on the receiving end of that, you know, after 30s, because of TikTok, my attention span ends and I need to talk about something else. And also, just the way that we do business now are in these short 30 minute meetings where you completely swap context. And so I think those are difficult challenges not related to AI that are still applying pressure on human attention span. But it has a pretty strong consequence on how people learn and how students can learn when they're constantly being distracted, when they struggle to sit with material. Over time, I think AI can perhaps assist in, resolving that by its ability to personalize the experience to the individual and engage them more effectively. And so if you have a generic education offering which, you know, bores some part of the population excites the other, you're missing you're underserving that that population that gets bored. But if we can have a very targeted, scalable approach to for each individual, giving them something that's engaging, exciting, if they are auditory learners or visual learners, we can tailor it to them and hopefully keep their attention better than we might otherwise would. So AI might be part of the solution as opposed to the source of the problem.
Yeah. You go. Does your vision of AI comport with that?
It completely matches. And I think, you know, there's a well-known piece of research from the 80s, from a University of Chicago professor. It's the Bloom Two Sigma problem. And they did some research where they looked at the ability to learn with one on one coaching. It was two sigma higher than the classroom. But the economics of doing that was not there. That's why we have these big classrooms. And that's why there are bigger classrooms for first years. And that's why it doesn't deliver the same learning experience. Now to Aiden's point with AI, you can personalize the experience, you can adapt it, and you can create feedback loops that a professor cannot. Today. You know, you've got 40 students. You cannot pick up who's you know, not easily. Some some teachers are amazing and they have the ability to to do incredible things. But now you have the ability to have that feedback. So I think, you know, we're going to see a lot of AI, expert tutors and coaches that will have context and that will have been trained on a body of knowledge that is hopefully trusted, hopefully accurate, and, will help, you know, in the way that you like to learn. So if you're in an auditory learner, we're going to give it to you that way. And if you're a visual will give it to you that way. I think that's that's a really exciting and promising, world we're entering from that point of view.
So we're going to go to questions from the audience, in just a second. So start thinking about your question. I'm just going to ask one more question of our panelists myself, which is, where do humans fit in? In, in in this brave new world of, of AI based education? I think all of us who are educators know that at some point we need human intervention in the in the process, even with the most fabulous technology. Where do you think they need to come in?
I think they're the customer. Right. So they're the ones that we're serving with this technology. And so we need to create the best possible product for them. If we just do surface level education, that's very confirmatory. Oh, yeah. You've got it. Great. You know, a bit sycophantic. Then they won't be effective in the real world when they actually enter the job market. And so there's a burden on us as product creators to create the most effective product, to teach people skills and give them knowledge. And I think that AI is actually an incredibly effective tool towards that. But I do still believe that it's a tool. It's like a calculator. It's something that you can lean on to give you faster answers, more thorough answers. But we still need to ground ourselves in the human without the tool. And so testing becomes it's always been important, of course, but I think it becomes absolutely critical now because you can fake your way through an education system much more easily. And so having very strict testing regimens is going to be essential.
Yeah, I have a variation on this. I do think, the teachers, the instructors are part of the partly the customers, but I do think they're, you know, they need to be in the loop. They're amazing storytellers. They have a way. I mean, if I ask anybody in this room who was your favorite teacher in high school? And I paused for five seconds. There's somebody in your mind right now. What was special about that person? And you cannot replicate that, but you can augment that. You can make that person now be able to maybe teach you on something that they were not that, you know, maybe like my favorite teacher in high school was a physics teacher. I loved the way he presented. I loved the way he engaged. And it was so motivating. My chemistry teacher was not that. But now I can augment, you know, with AI and have the voice. Not just the voice, but the way he thought, the way he presented the information be applied to a different topic. And I think that gets pretty, pretty exciting as well.
They understand chemistry.
I may finally understand chemistry. I stayed away from chemistry because of that. But physics I loved.
Okay, I want to I want to open up to questions from the audience. So I will call on you the old fashioned way. If you raise your hand. Yeah. Oh. You have. Sorry. You have to speak into.
Director of the Leverhulme Trust. I guess learning is a bit like working out. It's got to hurt to be effective. How do you think AI enabled tech? Various kinds can help with that motivation issue. You've talked about the teacher being the one. Absolutely. The motivates. But a lot of the systems we're talking about in the workplace, etc., you're not going to have that human in the loop. So can we do things with AI and tech that could prompt that?
Yeah, I'm going to offer a few a few suggestions. And this is not like future. This exists today. So you can do AI role play in a way that makes you go through, you know, the learning process. And I'm going to use Business Example. So if you're a a new salesperson and you have a new product that you need to sell, you can load up the specs of that product into an AI role play and practice selling to a person. And there will be a rubric against which we're going to score you, and we're going to discover whether or not you are competent at selling this product that you're responsible for. So that's a business example. I can do the same thing in a call center. You know, we have a 21 of the largest call center outsourcers. There are 20,000 call center agents. They need to onboard every, every month. That is incredibly complicated. But now you can load, you know, the most common error cause the most common tickets, the product specs. And instead of taking three weeks to onboard somebody through the process of learning, of experimenting, you can do a role play and get to, you know, accelerate that learning by doing a lot of practice. So it's simulation. So that's that's one powerful example. I think the other one is AI can give you feedback and monitor the progress you're making in the way that we can bring you back to that point in the gym where you're struggling with whatever exercise you're doing. We're going to make you do that exercise more and more and get that repetition in a way that reinforces the gap that you have.
There's one here.
Hi, I'm Nathaniel. I run an education company in Australia. Now, as a region, Australia has an interesting relationship with technology. As many of you may know, we've just recently had a social media ban for young people under 16. And in a similar vein, we don't really have a good consensus around the role of AI in classrooms. So my question is, what do you believe the role is for AI in physical classrooms? And what would you say to people who might be on the side of banning versus not banning it?
Yeah, I think I'm interested to hear your answer. But from my side, I think it's a tool, like a calculator. I think also a duty of the education system now is to teach people how to use this AI, how to engage with it, how to most effectively use that tool. And so it certainly should exist as part of the classroom and as part of schooling. But like I said, it can become a crutch and it can be used to cheat. And so we have to come up with ways to ensure that students aren't misusing it or using it in ways that are unproductive to their learning. I'm excited.
To hear you. I've got two part answer. The first one is, any, you know, business process or any, endeavor, you have the problem statement asking the right question. You have the solving, and then you have the quality assurance in the back. It's a feedback loop that you go through a circle all the time. And education is no different. What AI does well is that middle part. It doesn't do a whole lot in the front end and the back end. So what we need to teach young students and adults is how to ask the right question, the critical thinking. I love that it came out at the very top. Super super important. But you you can, you can. As you said, the calculator is a calculator. Like the fact that I can't do multiplication table all the way to 100 is not that relevant for my day to day job, but the fact that I can be critical in my thinking, I can summarize, I can contextualize. I think those are the skills you want. The second part, for those who are curious, I have no relationship, but I am just fascinated. There's a school in the US called Alpha School and they've got a really powerful model. They are using AI, they are encouraging students to use AI, and they're demonstrating that I'm going to get all the stats wrong, but they get two AIS, the learning and half the time or three times the learning half the time, and then the kids in the afternoon, they go learn and learn how to be a civic leader or a leader in all sorts of other contexts, instead of spending all their time where, you know, historically you would have learn, you know, various dates. It's not that relevant to know the dates of specific things, but it's relevant to understand the context of those events. And I think that's where we can focus a lot of the effort.
Okay.
Thank you. Terrific topic to be discussed at Davos I'm Pranjal Sharma I'm from India I'm an author and analyst. We're looking at a lot of the micro pieces. But I'd like to focus on the micro. We have a situation today where we are all skilled up but nowhere to go. Right. Last year I think ILO says 7 million fewer jobs were created, not to mention the existing jobs that disappeared. So there is a cry from the industry. Firstly, they don't know who to hire and why to hire and what to hire, and they don't even know what to test their credentials on. The second part is there's a huge disconnect between what they want and what academia is offering. Plus, the concept of a degree shouldn't exist and even continuous learning in terms of applied knowledge is missing. So I think the core phrase to be used here is applied knowledge. How do you create information for a person to be able to earn a livelihood irrespective of white, grey, blue collar? And I think that's the gap of applied knowledge delivered in the right way to the right people, at the right time.
From a labour market perspective, I think there's a, a good case to be concerned about the impact of AI and what might happen. And reskilling is going to be an essential component of that. The mismatch in the market between, what education institutions are offering and what the market is demanding. I think that is a major issue that we need to figure out how to solve. I think AI can be a part of speeding up delivery of new programs and courses, and keeping up with changes in demand much faster than we have in the past. The process of scaling up educational infrastructure to meet a shift in market demand has been historically extremely slow and laborious. But with AI, we're able to create programs much faster. The models are, you know, infinitely scalable. They're always awake 24 over seven. They're extremely they never get annoyed at the student. Right. So we have these incredibly compelling tutors to deploy at scale against the problem of teaching the population the skills that we need. But I think the issue might be in identifying the skills that we need. And that's still going to have to come from us, the humans, the business leaders, the the policymakers. So that might be the core constraint. We need a direction to be set against to start building the solution.
I think, to I mean, what I would say is I think that that, you know, universities, aren't teaching to what businesses need necessarily. They're, we're teaching things that we believe are fundamentally important. And I would defend that. I mean, we're teaching critical thinking and we're teaching deep mastery, and we're teaching them to people at a critical moment in their lives, most of them where they actually really need to have a go and and learn these skills. They may need additional skills when they go out into the workplace. And that, as far as I'm concerned, is what the kinds of products that you're talking about are for. So.
Yes.
Good. Thank you. Just go back to the critical thinking because, now, in the university, the students widely use the AI assistance and get the instant answer. In that case, how can we, teach them to increase their capability, of critical thinking, to make factual, check. Logical check. Scientific check. Ethical check to the instant answer they got from, arm models.
So you're at least. Yeah.
From my point of view, you're on to one of the core issues. We need to start teaching a very different set of skills. You need to, in my little model of, you know, ask the right question, get the answer, and then check the answer. The middle part AI is going to outdo the human like it is already a foregone conclusion. The AI will outdo the human. So where we can be competitively differentiated versus the AI is in the front and in the back end. So we need to adapt the curriculum to make sure that people are asking the right questions with the right context. And it is critical thinking, it is critical thinking, but we need to expand, and we need to have a better way to evaluate the level of critical thinking these students have when they hit the workforce, so that you can evaluate. And then the same on on assessing. I mean, AI is marvelous right now. It generates codes like there's no tomorrow, but it's mostly garbage. It is, you know, we have bottlenecks and quality assurance in the back end. So how do you kind of create the new tools and teach people to have the critical thinking to see if this is using the right library? Is using the right pattern? Is it using the right data? I think that's one of the core, you know, change that. You know, academic institution, organization like me, an individual need to do as you do your self-development, you need to kind of really lean into this, this ability to ask the right question, because the middle part, you don't have a competitive advantage, you will be outgunned. And the thing that is even more crazy is historically like people did. PhD I have a PhD. I went like super deep on one topic, and I got buried somewhere in the sinkhole, and it took my entire body of effort to get there. And to be a polymath is very hard to be able to understand. I don't I know nothing about chemistry. I know nothing about biology, psychology. My dad did that, so I got something rubbed up on me. Maybe. So. But AI is a polymath by design. It has the data set across all of that. So the middle part is a foregone conclusion. Folks, you need to get good at the front and the back end.
Yeah. Well, I was going to say another thing which is teaching is a skill. In the same way coding is a skill or doing math is a skill. And so it's a core capability that we as developers need to invest in. And, it's not something that is easily benchmarked. And it's not something that is accurately tracked at the moment. But I think the more this rolls out, I mean, it's already in the hands of every student on the face of the planet. It's going to become imperative that we're able to track the performance of models in teaching tasks to ensure that they're actually effective, and improve that over time. That's just so like a technical level that is not done presently. I don't know of a teaching benchmark, but I can point to probably 30 code ones, 50 math ones, you know, biology, etc., right?
Just just to follow up on this, I think, Mike, you made a very profound statement. Actually.
It happens from time to time.
I think that psychology is rubbing off. Well, when you say AI is a polymath by design, it's a brilliant thought. You know, it was you articulated it very well, which also means that by definition, humans cannot compete. So we basically have to end the session and say, the doom is nigh.
Well, I don't.
Think so. I mean, I'm more optimistic. So the polymath thing is real. I mean, you know, like if you do again, historical perspective, he who had Leonardo da Vinci on his team, you know, had an advantage to build War machine or, you know, a better court or whatever. Now there's going to be a similar debate, like who assembles these polymath AI thingy has an advantage, okay, that that is a foregone conclusion. That's why there's all these, you know, battle for, but I think we cannot, as the human race, give up that ability to influence. I think we made a point. I think you did at the very beginning. Like, these models typically are not designed, though some of them can be designed to explain their reasoning. So if as a society, we begin to rely on this thing that is super facile, that gives us an answer and we don't have the questioning, and we don't kind of do the checking and the validating, we lose agency on important decisions. And I think that is one of the things that we need to focus on deeply as a society. It's it also leads to the guardrail, the ethical things and all that other stuff. We need to go there because in the middle it's going to come up with answers that will be amazing in biology and will solve things in biology because it got trained in English language, I don't know, but it's going to be pretty, pretty wild. But we cannot lose agency around this polymath. I mean, every data center is going to have hundreds of millions of polymath in their. Yeah.
I just want to share a thinking, I believe there's a type of paradox within companies about this critical thinking. Let me let me say it this way. We see a lot of senior professionals. We know how to judge what the AI is doing. So I asked her one day for AI to do a model, whatever, and I could judge my juniors. They were not able to judge because they don't have the experience. So but to some extent I could fire them because I don't need them anymore because of this AI, technologies. But maybe there will be a gap. So at some point in time, AI can enhance a lot what I do. But if you don't train, let's say the new generation, the junior who will be the future, who will be in the future, able to to, do this critical thinking on what AI is doing. I don't have the answers. Obviously, companies need to take efficiency and we need to to do our best to, to reduce cost, whatever. But I think it's something we as, as a society, we have to think a lot about.
Yeah, it's fair.
We've got one here. You wanted you were up, right?
Yeah.
I didn't just cold call you.
Hi. Thank you for your insights. I'm. I'm Kian. I'm the CEO of an AI company called Workera. I really like what you said. Aiden, on, testing the human. And I think in in the world of testing right now, there's almost two camps. One that says you can test them with the calculator, we can test them without the calculator. And there's also overlaid on top of it the risks of proctoring and understanding, who's cheating, who's not cheating, and what can you tell about it? So how are you thinking about that idea of testing with or without the calculator?
Yeah, the, the like. Can you tell whether a piece of text was written by AI? It's really tough. A lot of the detectors out there are total scams. They'll say 100% AI even when it's not used at all. So they're extremely overconfident. Very high error rate on both sides. False positive. False negative. And but the answer to that question is you can like, you can insert into language models subtle cues to indicate for the reader this was written by an AI. You can not sample from natural language language that I'm drawing from right now. You can sample from a slightly shifted distribution and use certain words much more than any normal any human would use. And then as soon as those words appear, you have a good piece of evidence that this was written by a language model. And so us language modeling companies do that. We we shift the distribution of the language model so that when its text gets read, we have some ability, to say, you know, I can assign a likelihood that that was generated by my model. So you can detect that to some extent. But many of the tools are scams. And so I think we need to make better tools and put them in the hands of educators more readily. On testing with and without the calculator, I have a pretty strong focus on without the calculator. Like, I think everything needs to be ripped away and you standing alone as yourself, need to prove your knowledge. That is like the gold standard test of what you have, what you have learned retained. But of course, like I was saying earlier, using the language model is a skill itself, and we should have space to test that. In which case, of course, you're going to need the LLM in the loop.
Let me, let me sees the, the, chair's prerogative here to ask because I'm curious what you would both say, to this question, what happens in this, in this brave new world of, of polymaths and not showing your work and, not explaining your answer, to to expertise or authority? So, you know, there we have Cambridge, you know, library after library of big books that tell you the truth, or that that was always the, the the idea. Right. You would go look it up somewhere. What do you do in a world in which looking it up is no longer. There's not a dictionary, there's not a truth.
Well, I'll start. I think most technology go back and forth. There's a pendulum. We're in the pendulum. That bigger is better. We're throwing everything under the sun. Every Reddit quote is now part of training every large language model. And that is good. It's going to give you an average answer for average problem over time. I think we're going to come back and say you do need specialized trusted, and we need to have confidence that we did use the right source. And I think there will be a space for that. At least I want to hope that that will be the case, that we're going to come back and we're going to have these specialized model that will not only be Rag, but are going to be, you know, fine from from scratch with the right intent. And they don't need to be a bazillion trillion function points and whatever. I mean, they just need to be trained on the expertise. And you do need to trust it. And it's it's going to be incredibly important. I think we also need a lot of research on explainability. These and Bengio, University of Montreal, one of the guys who got the Turing Award, has been very vocal around this. We need to kind of go back and explain a lot more of these. These are statistical model. This is all this is. It's huge matrices. And they're like weights assigned to different things. So the people so this is not a piece of software where you say if then this that this is just statistics. So it on averages gives good answers. But it depends on the data. And you need to come back and put a bunch of tools to put the explainability into the model. And there are ways to do it. It's not yet super advanced and I think we need to invest in that so that we do, you know, have the confidence, build the trust. And I do think it's part of the learning question you have, because if the models are black box, you lose the ability to learn from their deduction process, which doesn't exist. It's just a statistical model. There's no deduction. So anyway, those are my two two ideas.
Yeah. Over the course of last year, there was a paradigm shift in the type of model that gets used. Now we don't just use, input output direct response models like you were alluding to. Every model now is a reasoning model. And so before it actually responds, it has an internal monologue where it thinks through the problem and tries to reason about it and then delivers response. It is primitive. It's a year old, but it's getting much better. And so I think exposing that to the user and showing these chains of thought, this reasoning is an important solution. And then like you say, Rag, which is retrieval, augmented generation, where the model isn't just drawing on its own knowledge, but it's actually making direct and specific reference to external knowledge. So we can plug it into the Cambridge Library. I went to Oxford, so the Bodleian, you know, but and it can cite directly back from, from those sources, and that that provides some degree of both reasoning and rag provide some degree of auditability. So you can have a little bit more confidence in the response because you can check its work.
Just out of curiosity.
What's driving that? What's what's driving the need.
For reasoning? Yeah, because the models were brittle. They would very confidently answer with the wrong solution. And it turns out, you know, humans don't put the same amount of energy into answering every question. But that was the previous the prior expectation on these models. You would ask them, what's one plus one? And it would immediately respond and put the same amount of effort into answering that question. And you would ask it, you know, to prove some unsolved er, dose problem or something. And it would put the same amount of effort as one plus one into that. That was obviously wrong. You know, there are some problems that we should spend days, weeks, months, years, decades, putting effort into solve. And there are others that can be responded to instantly. It's just a better, more robust intelligence.
That's fascinating.
We have time for one more question. Anything pressing? Let's.
Thank you.
Yeah, I'm a very interested to, ask the question of, just circling back to the beginning where we said we have like, public sector university as well as, a technology edtech platform being in the same room. The question I have in my mind is that, with right now, like in, in the US especially, education cost is so astronomically high and prohibitive. Lots of people are saying, the narrative goes is like, there's no point going to university anymore. And, I would see in that world there would be a lot of attention, turned to online education. I think we're all very familiar with Udemy. Has what is the gaps between, an online education and an accredited college or an elite college? Is there ever been, customer or market demand for, online education to move towards a model or, or or imitate, a traditional college experience, like, has that ever surfaced as a need? And, yeah, just like comparing the gaps there.
I'm going to say something maybe controversial, but it's fun. The university degree is a bundle. It's a convenient bundle that has a society which chose to create. So you learn something, you get an accreditation, and you have a rite of passage. You know, these kids are at the moment, they leave home, they go and they and that bundle is a convenient. Then we bundle that with research because the same people could now pass on their knowledge to others. It is a convenient bundle. As a society, it has worked well for, you know, a long time. Oxford and Cambridge are examples of long standing institutions that had a version of this bundle. It changes over time. Is it time to revisit whether all of these components need to fit together because of the economics and what AI can do to change the economics of delivery? Maybe. I think, the second.
I think it quickly.
Yeah, quickly, yeah. And the second piece is just the adaptability. If you have the labor market that moves so fast, you're now going to begin to put more weight on addressing a specific need for a specific skill. So I think that is a reality. In addition to that potential unbundling of that whole experience.
You have a good word for the for the for the university.
I don't I'm actually interested to hear from from the university's perspective.
Then I'll then I'll just. Our university does so much more than provide knowledge that it still is is worth its weight in in gold. And it is gold. But but you know, we'll see how the space develops, right? And with that, I'm getting all kinds of signals from the producers, so we've got to end it. But thank you very much. Thank you for your questions. And thank you to our panelists.
Thank you. Thank you.